P-values

What does a p-value mean?

What does a p-value mean?

What does a p-value mean?

• In the broadest sense, p-values indicate the probability of getting a result >, <, or absolutely greater than as the one in your sample if the null hypothesis is true.

• This is a statement about the probability of a data given a model, not a statement about the probability of a model.

  • In frequentist statistics, parameters are fixed and hypotheses are either true or false, so it doesn’t make sense to talk about the “probability of a model”.

P-values and Type I error

.05 level means that around 1 in 20 of results would be false positives

What does a p-value not mean?

• \(P(t | h_0) \neq P(h_1)\) P values are not the probability of the alternative

Regression p-values

In regression output, the p-value typically indicates the probability of seeing a T-statistic as extreme as the one in your sample if the null hypothesis is true:

voteshares$income_over_65k<- voteshares$median_income>65000
gof_stats <- c(
  "nobs"
)
coef_names <- c("income_over_65kTRUE" = "Median income over >$65K"
                )
model_1 <-lm(demshare ~ income_over_65k, data=voteshares)
modelsummary(list("DV: 2020 statewide Biden %" = model_1), 
             statistic = c( "p.value"),
             gof_map = 'nobs',
             coef_rename = coef_names,
             notes = "Note: p.values in parentheses"
             )

Importantly, this assumes that the sampling distribution of coefficients is approximately normal.

Regression p-values: randomization inference

There are some methods for relaxing this assumption, however. One method, mostly used for experiments, is randomization inference, where a binary treatment is randomly “shuffled”:

model_1 <-lm(demshare ~ income_over_65k, data=voteshares)
obs_effect<-model_1$coefficients[2]

set.seed(888)
perms<-50000
effects<-numeric(length = perms)
for(i in 1:perms){
  permuted <- sample(voteshares$income_over_65k)
  mod <-lm(demshare ~ permuted, data=voteshares)
  
  est_effect<-coef(mod)[2]
  effects[i] <- est_effect
  
}

mean((effects >= obs_effect) | (effects <= obs_effect * -1)) * 100

[1] 0.008

Regression p-values: randomization inference

Confidence intervals: bootstrapping

Another non-parametric method is the bootstrap, which relies on re-sampling observations with replacement and then re-running the regression model.

• Although the more common and intuitive application is to use the bootstrap to calculate a confidence interval…

model_1 <-lm(demshare ~ income_over_65k, data=voteshares)

iterations<-10000
results<-vector('list', length=iterations)
model_data<-model_1$model

for(i in 1:iterations){
  booted<-model_data|>slice_sample(n = nrow(model_data), replace=T)
  
  mod <-lm(demshare ~ income_over_65k, data=booted)
  results[[i]] <- tidy(mod)
  
}
result_frame<-bind_rows(results, .id='iteration')
result_frame|>
  group_by(term)|>
  summarize(
    lb = quantile(estimate, .025),
    est = median(estimate),
    ub = quantile(estimate, .975),
    p = min(mean(estimate>0), mean(estimate<0)) * 2
    )

::: :::::

P-values: caveats

• P1. p-values can indicate how incompatible the data are with a specified statistical model.
• P2. p-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
• P3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
• P4. Proper inference requires full reporting and transparency.
• P5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
• P6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis

From: McShane, B. B., & Gal, D. (2017). Statistical significance and the dichotomization of evidence. Journal of the American Statistical Association, 112(519), 885-895.

Problems with P-values

• Often misinterpreted, even by experts

• They’re continuous, but they tend to be treated as binary and conflated with importance

• In large samples, they’re hard to read because they get infinitesimally small

Confidence intervals

What do confidence intervals mean?

In repeated sampling, 95% of the 95% confidence intervals will contain the correct value of \(\beta\)

Confidence Intervals

• Its misleading to say there’s a 95% chance that the population \(\beta\) falls between the ci boundaries. (frequentist stats assumes that \(\beta\) is fixed, so it doesn’t have a “chance” of doing anything.)

• Better to say “I’m 95% certain this range contains the true value”

Confidence Intervals

• Pro: P.values and Confidence Intervals are based on the same basic assumptions, but CIs can be more informative about uncertainty and don’t lend themselves to the same dichotomous interpretation that a p.value does.

• Pro: Automatically scaled to the quantity of interest, so harder to confuse with a measure of substantive importance

• Pro: can be used for equivalence testing (we’ll come back to this)

• Con: also commonly misinterpreted!

Alternatives: S-Values

Alternatives: S-Values

• Pro: Still based on the same assumptions as a p-value, its just a transformation of the p.value that might be more intuitive for people.

• Pro: easy calculation (for a computer, at least)

• Con: Not widely adopted. All of this stuff relies on convention.

Bayesian methods

• Bayesian methods potentially allow us to do the thing we wish we could do with p-values: quantify hypothesis probabilities.

• One downside is that it sort of requires everyone to adopt a totally different philosophical approach to probabilities: Bayesian statistics assume that the model parameters also have a distribution of their own, and we use data to update our knowledge about the nature of those distributions.

• Another downside is that Bayesian inference requires us to come up with “prior” distributions for model parameters, which invariably involves some subjectivity.

Bayes Factors Approximation

Bayes Factors measure the weight of evidence in favor of one hypothesis relative to another.

Wagenmakers (2007) suggests a frequentist approximation of a Bayes Factor using the Bayesian Information Criterion.

• Run a model with the variable of interest (model 1)

• Run a model without the variable of interest (model 0)

• Calculate the BIC for both

• Calculate \(\frac{\exp(\text{BIC}_1)/2}{\exp(\text{BIC}_0)/2}\)

Bayes Factors Approximation

ff<-read_csv("https://vincentarelbundock.github.io/Rdatasets/csv/causaldata/gov_transfers.csv")

model_0<-lm(Support ~  Income_Centered + Education + Age, data=ff)

model_1<-lm(Support ~  Participation + Income_Centered + Education + Age, data=ff)





exp(-BIC(model_1)/2)/exp(-BIC(model_0)/2)

[1] 7.509412

Numbers greater than 1 indicate evidence in favor of model 1, numbers less than 1 indicate evidence against model 1.

Comparing Models

• If I have multiple models, how do I tell whether one is better than the other?

“Good” models can be bad predictors

R-squared

R-squared can be used for model comparison, but it doesn’t tell you a model is a good predictor.

Comparing models

• Make a fair comparison: our “restricted” model \(Y_i = \bar{Y} + e_i\)

Comparing models

Predicting Democratic voteshare using the 2016 share

Comparing models

\[ F = \frac{(RSS_{restricted} - RSS_{full})}{df_{restricted} - df_{full}} \div \frac{RSS_{full}}{df_{full}} \]

model_0 <- lm(demshare~ 1, data=voteshares)

model_1 <-lm(demshare ~ 1 + demshare_2016, data=voteshares)

anova(model_0, model_1)

Comparing models

We can continue to include additional covariates in the model. But the “fair comparison” here is probably the same model -1 covariate (rather than comparing to the null model)

BIC and AIC

• BIC stands for Bayesian Information Criteria.
• AIC stands for Aikake Information Criteria.
• Both values are based on the model likelihood, which, for linear models, is based on the sum of squared residuals:

BIC and AIC

Lower values of both AIC and BIC indicate better model fits:

BIC(model_1)

[1] -284.4724

BIC(model_2)

[1] -281.4003

AIC(model_1)

[1] -290.2679

AIC(model_2)

[1] -289.1276

Out of sample prediction

• Basic method: set aside some observations, create model on the training set and then predict the test set

• For small samples, K-fold or LOOCV are preferable: leave out one, predict, and then do it again. Then average the metric.

library(caret)
# Define training control
train.control <- trainControl(method = "LOOCV")
# Train the model
cv_1 <- train(demshare ~ demshare_2016, data = voteshares, 
               method = "lm",
               trControl = train.control)

Out of sample prediction

Linear Regression 

51 samples
 1 predictor

No pre-processing
Resampling: Leave-One-Out Cross-Validation 
Summary of sample sizes: 50, 50, 50, 50, 50, 50, ... 
Resampling results:

  RMSE        Rsquared   MAE       
  0.01380351  0.9871893  0.01096933

Tuning parameter 'intercept' was held constant at a value of TRUE

Out of sample prediction

…but is this the right metric?

Arguing for a null

Conventional hypothesis testing “stacks the deck” in favor of finding no relationship, and failure to reject the null could simply be a function of small sample sizes.

Still, we often want to dismiss a claim:

• Can citizen pressure campaigns make states more effective?

• Do motor voter laws improve turnout?

• Are some people actually psychic?

TOST and equivalence testing

“Two one-sided tests”

Step 1: Set a minimum effect size (\(\Delta\)) worth caring about

Step 2: Conduct one-sided t-test for:

• H01: \(\Delta \leq −Δ_l\)

• H02: \(\Delta \geq Δ_u\)

If both of these tests are rejected, then we can conclude the observed effect is smaller than the minimum effect size.

TOST and equivalence testing

TOST and equivalence testing

Moery and Calin-Jageman’s replication results look like this:

TOST and equivalence testing

The estimated effect: 5.25 - 5.22 = 0.03

Minimal effect size \(\Delta = 0.43\)

TOST and equivalence testing

There’s also an equivalent method that relies on 90% confidence intervals instead of T.tests:

Null effects and Duverger’s law

Rainey:

• “Hypothesis 1: Increasing the effective number of ethnic groups from the 10th percentile (1.06) to the 90th percentile (2.48) will not lead to a substantively meaningful change in the effective number of political parties when the district magnitude is one.”

• “Hypothesis 2: Increasing the district magnitude from one to seven will not lead to a substantively meaningful change in the effective number of political parties when the effective number of ethnic groups is one.”

• Substantively meaningful effect: changing the effective number of parties by 0.62 (this is the difference in the effective number of political parties between the U.S. and the U.K.)

Null effects and Duverger’s law

Marginal effects and quantities of interest

• For simple regression models, we can get most of the interesting stuff from just looking at the output, but what if we’re working with something more complex?

For instance, this model includes a polynomial term for percent Bachelor’s degrees in a given state. What is the effect of a one unit increase in Bachelor’s degrees, and is this statistically significant?

Call:
lm(formula = demshare ~ poly(pct_bachelors, 2, raw = T), data = voteshares)

Residuals:
      Min        1Q    Median        3Q       Max 
-0.188633 -0.071404  0.007378  0.046200  0.214193 

Coefficients:
                                  Estimate Std. Error t value Pr(>|t|)  
(Intercept)                       0.595663   0.391311   1.522   0.1345  
poly(pct_bachelors, 2, raw = T)1 -0.057409   0.058086  -0.988   0.3279  
poly(pct_bachelors, 2, raw = T)2  0.003591   0.002125   1.690   0.0975 .
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.08616 on 48 degrees of freedom
Multiple R-squared:  0.5299,    Adjusted R-squared:  0.5103 
F-statistic: 27.05 on 2 and 48 DF,  p-value: 1.356e-08

Marginal effects and quantities of interest

There’s not a straightforward way to calculate an effect or a confidence interval here analytically, so we need to use other methods.

One option is bootstrapping, another is simulation.

      2.5%        50%      97.5% 
0.03353758 0.04420986 0.05611655 

The marginal effects package will do this automatically:

 Estimate Std. Error    z Pr(>|z|)    S  2.5 % 97.5 %
   0.0441    0.00606 7.28   <0.001 41.4 0.0322  0.056

Term: pct_bachelors
Type: response
Comparison: +1

Marginal effects and quantities of interest